Notes for Lecture 17 Log 1 1 −

In the previous lecture, we introduced the notion of a randomness extractor —a procedure which extracts " uniform " randomness from a weak random source and a small number of truly random bits. In this lecture, we give general impossibility results for randomness extraction. The results are taken from [NZ96, RT00]. We seek to characterize the class of random sources for which extraction is possible. We start with a discussion of Shannon's Entropy. We explain, in particular, why this notion is not appropriate for our purposes. P[X = x] log 1 P[X = x] = E x∼X log 1 P[X = x]. An interpretation of H(X) is as follows: it counts the number of truly random bits needed to sample from X. Indeed, if t bits are used to encode a particular value a, then we need P[X = a] ≥ 1 2 t ⇒ t ≥ log 1 P[X = a]. Therefore, if R a denotes the number of random bits used for a, then we have E a∼X [R a ] ≥ E a∼X log 1 P[X = a]. However, Shannon's Entropy measures only the amount of randomness needed, not how it is " distributed " over all possible values. To see why this is a problem, consider the following random source X over {0, 1} n. Suppose X is such that P[X = 0] = 1 − 1 2 100 , where 0 = (0,. .. , 0), and for all a = 0, P[X = a] = 1 2 100 1 2 n − 1 ∼ 1 2 n+100 .